graph LR
A[Simple<br/>Hypotheses & Methods] --> B[Detailed<br/>Analysis Plans] --> C[Registered Reports<br/>Peer Review]
A --> D[Easy]
B --> E[Medium]
C --> F[Difficult]
Enhancing Transparency and Reproducibility in Language Science
2025-07-16
Goal: Make your language research more credible and transparent
Roettger, T. B. (2021). Preregistration in experimental linguistics:
Applications, challenges, and limitations.
Linguistics, 59(5), 1227–1249.
Why this paper?
Key takeaways to focus on:
Preregistration refers to posting a timestamped outline
of the research questions, hypotheses, method, and analysis plan
for a specific project prior to data collection and/or analysis
Key principle: Distinguish between:
graph LR
A[Simple<br/>Hypotheses & Methods] --> B[Detailed<br/>Analysis Plans] --> C[Registered Reports<br/>Peer Review]
A --> D[Easy]
B --> E[Medium]
C --> F[Difficult]
Preregistrations can vary from simple outlines to comprehensive analysis plans with pre-written code
Result: Scientific record biased toward positive findings
Consequence: False positives may mislead theory development
Recent findings from our field:
Evidence: Roettger (2021) documents widespread analytical flexibility in linguistics
Reference: Roettger, T. B. (2021). Preregistration in experimental linguistics:
Applications, challenges, and limitations. Linguistics, 59(5), 1227-1249.
Key insight: Preregistration is flexible and adaptable to linguistic research
Goal: Be specific enough that a skeptical reader is convinced you planned ahead
🔗 Resource: OSF Templates
Linguistics-specific: Secondary data template
Focus on the essentials:
Good for: Beginners, exploratory studies, time constraints
Include specifics:
Good for: Confirmatory studies, complex designs
Two-stage process:
OSF (Open Science Framework) - Most comprehensive - Multiple templates - Integration with project management - Embargos up to 4 years
AsPredicted - Simple and quick - 8 basic questions - Good for beginners - Free to use
| Feature | OSF | AsPredicted |
|---|---|---|
| Templates | Many | One (9 questions) |
| Output | Web page | PDF with URL |
| Embargo | 4 years | Private option |
| Collaboration | Multi-author | Email approval |
| Cost | Free | Free |
| Integration | Project management | Standalone |
gantt
title Preregistration Timeline
dateFormat YYYY-MM-DD
section Planning
Draft preregistration :a1, 2025-01-01, 2w
Advisor review :a2, after a1, 1w
Revisions :a3, after a2, 1w
section Execution
Register study :b1, after a3, 1d
Data collection :b2, after b1, 8w
Analysis :b3, after b2, 4w
When things don’t go as planned:
Remember: Transparency is the goal, not perfect adherence
Standard language for reporting deviations:
Key principle: Transparency, not perfection
Scenario: You preregistered a study but encountered problems:
Discussion: How would you handle each?
The 9 AsPredicted Questions:
Result: Time-stamped PDF with unique URL for verification
Research Question: How does prosodic prominence
affect syntactic processing in German?
Hypothesis: Prominent words will show faster integration into syntactic structure
Participants: 40 German native speakers, 18-35 years, no language disorders
Materials: 120 sentences with prominence manipulation, normed for frequency/length
Procedure: Self-paced reading + comprehension questions
Analysis: Linear mixed-effects models with prominence as fixed factor
Exclusions: Accuracy <80% on comprehension, reading times >3 SDs
Research Question: Does word predictability
affect pronunciation in spontaneous speech?
Data: HCRC Map Task Corpus (Anderson et al., 1991)
Preregistered decisions: - Predictability measure: Trigram probability from Google Books - Acoustic measure: Mean F0 of vowel nucleus - Control variables: Speaker sex, utterance position, word frequency - Exclusions: Function words, words <3 phonemes - Model: Linear mixed-effects: F0 ~ predictability + controls + (1|speaker)
Key insight: Even with existing data, many analytical choices remain
Roettger’s key insight: “Preregistration is not a panacea for all problems,
but it’s a practice we can integrate into our work flow right away”
Immediate action: Choose one upcoming study to preregister
Essential websites:
Reading recommendations:
Small group task (10 minutes):
Scenarios provided:
What challenges do you see for preregistering linguistic research?
How might preregistration help your current SFB 1252 project?
Who in your research area could be your accountability partner?
Contact: job.schepens@uni-koeln.de | Project S, SFB 1252 Workshop Materials: Available on SFB 1252 OSF project
RDM Workshop - SFB 1252 Prominence in Language